1. Field of the Invention
The present invention relates to the protection of computer systems. More particularly, the present invention relates to a system and method for detecting and recovering from a stack frame corruption.
2. Description of the Related Art
Buffer overflow techniques have been used by malicious hackers and virus writers to attack computer systems. Buffers are data storage areas, which generally hold a predefined amount of data. A buffer overflow occurs when an application attempts to store data into the buffer, where the data is larger than the size of the buffer.
One category of buffer-overflow sometimes called stack-based buffer overflow, involves overwriting a stack memory, or stack. The stack is basically an area in memory at run-time that an application uses to store data temporarily.
A stack-based buffer overflow is typically caused by exploitation of a function that does not verify the length of the data being copied into a buffer. When the data exceeds the size of the buffer, the extra data can overflow into the adjacent memory locations in the stack. In this manner, it is possible to corrupt valid data and possibly to change the execution flow. Thus, by exploiting a buffer overflow, it is possible to inject malicious code into the execution flow of an application.
To detect corruptions of the stack, some prior art detection systems placed a canary value before each stack frame of an application's functions in the stack. The canary value was saved at each prologue of a function and checked at each epilogue of a function. If a change was detected, the application was terminated or a security handler was called.
Thus, these prior art detection systems were designed to detect stack frame violations and terminate the application. Consequently, the availability of the application was lost until the application could be restarted, typically detrimentally impacting system performance. Additionally, as these prior art detection systems were compiler-based solutions, the source code of the application was required or the vendor compiled the application with the stack frame violation detection.